Allow Whisper decoding to use the full decoder context - #2075
Conversation
|
Hi @jordimas, when you have a chance, would you mind taking a look at this PR? I’d really appreciate any feedback. Thanks! |
|
@Emre-Akgul please add a regression test covering this behavior: no-timestamp decoding can exceed the previous 224-token cap, while timestamped decoding keeps the existing limit. |
|
@MahmoudAshraf97 @Purfview see in case you have any comment |
|
The previous 224 token cap was in the reference open ai implementation, the model itself tolerates up to 448, that includes both input and output tokens, so as long as the total token count does not exceed that it's fine, note that whisper passes 4 tokens + the previous transcription, which are included in the 448 |
Covers the behavior requested in review: no-timestamp decoding can exceed the previous 224-token cap (up to 445), while timestamped decoding keeps the existing 224-token limit.
|
@MahmoudAshraf97 Thanks for the correction, you're right that the 224 cap isn't tied to timestamps. Updated the PR description's "Root Cause" section to reflect that it's just half of n_text_ctx inherited from OpenAI's reference implementation, not a timestamp requirement. The fix stays within the 448-token total budget you described, just uses what's left of it instead of an unconditional half. |
|
@jordimas Added a regression test (test_transformers_whisper_no_timestamps_full_context in python/tests/test_transformers.py) covering exactly that: no-timestamp decoding reaches 445 positions, timestamped decoding stays capped at 224. Verified it fails against the pre-fix code and passes with the fix. Kept timestamped decoding untouched since it's out of scope for the PR. |
This is inconsistency, since the limit was not related to timestamps, why is that separation still there? |
|
As I remember, length limits are handled at the faster-whisper level, why not just do: decoding_options.max_length = total_max_length - start_step;And use But I wonder why there was 224 cap in the first place? |
OpenAI implementation has it to accommodate the previous transcription prompts because it always truncated the prompt to 224 tokens which is half |
Emre-Akgul
changed the title
The half-context cap removed for no-timestamp decoding was never actually tied to timestamps - it originated as half of n_text_ctx in OpenAI's reference implementation and applied regardless of the <|notimestamps|> prompt token. Timestamped decoding now also uses the full remaining decoder context (total_max_length - start_step). Verified on large-v3 with the Armenian/Georgian samples from the previous commit using timestamped prompts: Armenian now completes fully (304 tokens vs. the previous 224-token cutoff), and Georgian reaches the full ~445-token decoder budget instead of stopping at 224.
b241df9 to
6d8b164
Compare
Applied the same fix to timestamped decoding in 6d8b164: both paths now use total_max_length - start_step instead of the /2 cap. Verified on large-v3 with the same Armenian/Georgian samples, now with timestamps enabled — Armenian goes from cutting off at 224 tokens to completing fully at 304 tokens (ending exactly at the audio's real timestamp), and Georgian reaches the full ~445-token budget instead of stopping at 224. Details and before/after examples for both modes are in the updated PR description. |
Even with condition_on_previous_text=False, the CTranslate2 code still capped generation at 224 tokens. The cap lived entirely inside ctranslate2's C++ code, not in faster-whisper's Python layer, so nothing faster-whisper does with its prompt/conditioning logic could route around it. |
Sure, probably it wasn't clear but that's what I meant. Could you check whether the PR have any speed impact on difficult audio/languages with lots of non-speech? I'm thinking of something like 70 years old Icelandic audio for example. Of course, this should be tested without VAD. |
The without_timestamps local was introduced to share a condition with the max_length ternary, which the previous commit already removed. Its only remaining use (gating the ApplyTimestampRules logits processor) is unrelated to this PR's max_length fix, so revert that check back to its original inline form per review feedback.
You're right. Reverted the without_timestamps refactor. On the speed question: for audio where the decoder naturally emits <|endoftext|> before 224 tokens (the common case), this change has zero effect — the old and new caps are both irrelevant since decoding stops earlier either way. The case where it matters: long, non-speech audio without VAD where the decoder hallucinates/repeats instead of hitting EOT naturally. There, the cap now allows ~2x more tokens before stopping, which means a slowdown on that specific pathological case. I haven't benchmarked it because I got no data at hand. |
Here is an audio for a test: https://www.transfernow.net/dl/20260830loYkBzko |
Ran the benchmark.
So the tradeoff comes down to this: for underrepresented languages that lean hard on the byte-level fallback tokenizer, the old 224-token cap could cut a correct transcript off mid-word, so the fix gets those languages a complete result. But when it hallucinates instead, you wait ~1.5x longer and get ~2x more junk output tokens for it. |
Huge speed diff, but it was expected in such edge case. Thanks for the test. Could you run one more test using non-difficult audio (a book)? |
4 participants
Summary
Whisper decoding unconditionally limited generation to half of
max_length, regardless of whether timestamps were enabled. This PR removes that half-context cap for both no-timestamp and timestamped decoding, letting each use the full decoder context that remains after the prompt.This improves transcription completeness for languages that rely heavily on Whisper's byte-level fallback tokenizer, where relatively short audio can consume a large number of decoder tokens.
Fixes #2074, #1856.
Root Cause
Whisper generation set:
decoding_options.max_length = std::min( total_max_length / 2, total_max_length - start_step );total_max_length - start_stepis the real safety bound — it's what actually keepsstart_step + max_lengthfrom exceeding the model's fixed 448-token context (n_text_ctx), and it already existed independently of the/2term. The/2term is a second, stricter ceiling stacked on top of it.Change
Both decoding paths now use the entire remaining decoder context:
With the standard 3-4 token prompt, decoding can now use up to ~445 positions instead of the previous 224, for both no-timestamp and timestamped generation.
Does this risk overflowing the 448-token context, or otherwise change output on cases that already worked? No.
total_max_length - start_stepis unchanged from before — it already guastart_step + max_length == total_max_lengthexactly, in both the old and new codeonly make the allowed length smaller-or-equal to what that bound already permitted,never larger, so there's no new overflow path. And it has no effect at all on audio that naturally finishes (emits<|endoftext|>) before hitting 224 tokens, which is the common case for typical-length audio.Reproduction
Both reproductions use a direct, single-window call (`ctranslate2.models.Whisper.ge the standard 30s/3000-frame window).
Model: Systran/faster-whisper-large-v3, same for both modes. Armenian and Georgian rely heavily on Whisper's byte-level fallbacktokenizer, so even relatively short audio can exhaust the previous 224-token limit — the Armenian reference needs 285 text tokens, the Georgian reference needs 516 (more than the model's complete 448-token context can ever hold, even after this fix).
No-timestamp decoding (
without_timestamps=True)Armenian — before fix (224 tokens, cuts off mid byte-sequence):
Armenian — after fix (261 tokens, completes normally):
Georgian — before fix (224 tokens, cuts off):
Georgian — after fix (445 tokens, uses the full decoder budget, still cuts off — the reference needs 516 tokens, more than the model's 448-token context can ever hold):
Timestamped decoding (default, no
<|notimestamps|>)Armenian — before fix (224 tokens, cuts off mid word):
Armenian — after fix (304 tokens, completes fully — decoding stops exactly at the|>`):
Georgian — before fix (224 tokens, cuts off):
Georgian — after fix (446 tokens, reaches the full decoder budget — still cuts off for the same reason as the no-timestamp case above):
(The timestamped Georgian result reaches 446 rather than 445 because its prompt is -timestamp prompt — it omits
<|notimestamps|>— leaving one extra position ofcontext.)Interpretation
The Armenian sample demonstrates the intended fix clearly in both modes: the transcription needs more than 224 tokens but fits within the remaining decoder context, so it now completes normally
instead of stopping mid-byte-sequence.
The Georgian sample exceeds the model's complete 448-token decoder context even remains truncated, but the decoder now returns substantially more of the transcription than the artificial 224-token cap previously allowed.
Validation
test_transformers_whisper_full_context,whisper-tiny) update and timestamped decoding now reach the full remaining context (445 / 446 positions)instead of the old 224 cap for timestamped decoding.large-v3decoding validated end-to-end on the Armenian and Georgian samtamp and timestamped decoding.